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ABSTRACT 

A study investigated 42 native Engl ish~speakers ' 

(NSs) perceptions of the pragmatic appropriateness of refusal 
statements. The NSs rated the appropriateness of 24 written 
statements in 4 different refusal scenarios, which were collected 
from both native speakers and non-native speakers. Pour weeks later, 
as a reliability check, the subjects rated the same statements again. 
Results indicate that what one subject considered pragmatically 
appropriate tended to be considered the same by other speakers of the 
same language. Also, the stronger the pragmatic impression, the more 
extreme the ratings, and the higher the level of rating consistency 
for a statement. In addition, subjects' pragmatic judgments tended to 
be consistent over time, and statements made by native speakers of 
English were considered pragmatically more appropriate than 
statements made by non-^native speakers as judged by native raters. 
Appended materials include the questionnaire used and data charts and 
summaries. Contains 24 references. (MSE) 
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;e^BCh has compared native and non-native pragmatic patterns from various 

[ves, using multiple methods. No one, ho^veve^, seems to have turned to the speaker of 



the language and ask him/her directly how do you FEEL about the way the f. atement is made ? 
Given the belief that pragmatics is organic and intuitive, the pragmatic appropriateness of a 
statement is most precisely determined by native speakeis intuitively and holistically rating it. In 
tuis study, 42 native speakers were asked to rate the pragmatic appropriateness of 24 written 
statements in four different refusal scenarios, v/hich were collected from native and non-native 
speakers of English (Chen, 1991). Four weeks later, as a reliability check, the subjects rated the 
same statements a second time. 

Results show that 1) what one considers pragmatically appropriate tended to also be 
considered the same by other speakers of the same language. Also, the stronger the pragmatic 
impression, the more extreme the ratings, iind the higher the level of rating consistency for a 
statement; 2) subjects’ pragmatic judgements tended to be consistent over time; and 3) statements 
made by native speakers of English were considered pragmatically more appropriate than 
statements by non-native speakers as judged by the native raters. 
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Native-Speakers’ Intuition: 



Rating Pragmatic Appropriateness 



1. INTRODUCTION 



Applied linguists distinguish non-native speakers’ pragmatic competence from their 
granunatical conpetence; pragmatic competence relates to the ability to function socioculturally 
felicitously, while grammatical competence alludes to language abilities such as syntactical and 
phonological competencies. It has been understood that non-native speakers’ pragmatic 
competence does not always develop with the acquisition of their grammatical competence. 
Hartford and Bardovi-Harlig (1992a) indicate tliat being able to function linguistically (i.e. 
grammatically) in a foreign/second language (L2) does not necessarily mean being able to use the 
target language in a socioculturally appropriate manner. Because of the discrepancy between the 
two types of competencies, nonnative speakers, even those who are highly linguistically proficient, 
often do not know how to follow context-specific, sociocultural rules; they may make a statement 
that is perfectly granunatical but socioculturally inappropriate under a given situation or fail to 
comprehend what is pragmatically meant by what is literally said in cross-cultural interaction and 
therefore result in pragmatic inappropriateness, or pragmatic failure (Thomas, 1983). 

Speech acts (e.g., apologies, refusals, appreciations, compliments, etc.,) since they are 
often socioculturally and sociolinguistically embedded, have been considered good ways to gain 
insight into one’s pragmatic competence and have, therefore, been studied by applied linguists and 
L2 educators. Speech act data gathering methods that are frequently used in research include 
natural data observation, role play, and Discourse Completion Tasks. Since these methods have 
varied limitations, this study employed a different data gathering method by asking 42 
undergraduate native speakers of English to holistically rate on a five-point Likert scale the 
pragmatic appropriateness of refusals carried out by both native and non-native speakers of 
English. The assumption of this study is tliat judgement of pragmatic appropriateness is intuitive 
and thus can be best projected by native speakers of the language reacting intuitively and 
holistically to speech act statements. The rationale is to investigate 1) whether what one considers 
pragmatically appropriate is also considered by other speakers of the same language to be equally 
pragmatically appropriate, 2) whether one’s pragmatic judgement is consistent over time, and 3) 
whether statements made by native speakers tend lo be considered pragmatically more appropriate 
than statements by non-native speakers as judged by native raters. Results in general indicate that 
one’s pragmatic judgement tended to be consistent over time, that there was a high consistency of 



4 



O 

ERIC 



Metapragmatic Judgement on Refusals 

4 









pragmatic judgement across raters speaking the same language, and that pragmatic ratings were 
higher for native statements than for non-native ones. 




2>1 Pragmatic Competence 

2.1.1 Form vs. Meaning/Function 

Leech (1933) divides linguistics into “grammar” and “pragmatics.” The former refers to 
the decontextualized formal system of language, while the latter alludes to the use of language in a 
goal-oriented speech situation in which the speaker uses language to produce a particular eff:ct in 
the mind of the hearer. Similarly, Thomas (1983, p. 92) considers linguistic competence to be 
made up of two types of competencies: granmiatical competence (including the abstract or 
decontextualized knowledge of intonation, phonology, syntax, semantics, etc.) and pragmatic 
competence (i.e. the ability to use language effectively in order to achieve a specific purpose and to 
understand language in context). As the grammatical part of language is associated with the form 
of language whereas the pragmatic part is concerned with meaning/function, and as one form may 
cany several levels of meanings or serve more than one function, a mismatch between form and 
meaning/function results in a situation in which the “sentence meaning” differs from the “speaker’s 
meaning” (Leech, 1983), or what Thomas calls “pragmatic failure” (1983). 

2.1.2 Pragmatic Failure 

Pragmatic failuie is not necessarily a lesuli of grammatical deficiency of the speaker; in 
Hartford and Bardovi-Harlig’s study (1992a), pragmatic failure is obseiwed in grammatically 
highly proficient non-native speakers of English. Rather, pragmatic failure is considered to result 
from the differences between languages in social rules of speaking (Hymes, 1972; Wolfson, 

1983b) and from “pragmatic transfer” (Leech, 1983), when non-native speakers transfer the 
pragmatic norms of their community to the use of the taiget language. Given a universal principle 
of speaking across all languages, there exists a culture- and language-specific pattern of realization 
in each language. Take, for example, the principle of “face” (Brown and Levinson, 1978). In 
communication and interaction, two aspects of people’s feelings are involved with “face.” One is 
the desire of the individual “not to be imposed on,” which is the “negative face,” and the other, the 
“positive face,”’ is the desire of the individual “to be liked and approved of.” It is universal that all 
languages observe this “face” principle by saving both the speaker’s and the listener’s positive or 
negative face. The actual way in which this “face” principle is localized or realized is language 
specific. Pragmatic failure tends to occur when a speaker goes across languages to observe a 
universal principle in a non-native language which realizes the principle very differently from his or 
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her native language. In observing the “face” principle, for instance, one language may require a 
degree of direc^ss much higher or lower than another language does. Therefore, a speaker, in 
preserving may fail to notice the difference in the degree of directr. ‘ss in the two languages 

imd too direct or indirect in the other language. 

2.2 Speech Acts 

Like the aforementioned principle of “face”, speech acts, although they exist universally in 
all languages, can be highly culture- and language-specific due to the different ways they are 
carried out in different cultures or languages. Consequently, in investigating cross-cultural, non- 
native pragmatic competence, researchers often focus on speech acts and compare native and non- 
native speech act patterns. Schmidt and Richards { 1980) maintain that “speech act theory can 
contribute to our understanding of second language acquisition” (p. 141). Hatch (1992) also 
alludes to the pedagogical implication of speech act research, "speech act theory has led to the 
design of the notional-functional syllabus in language teaching, setting off a major change in 
language teaching methodology ~ away from an emphasis on the linguistic form to an emphasis on 
language use” (p. 136). 

Much research has been done on major speech acts and various factors (e.g. familiai-ity, 
gender, and relative social status of the interlocutors) have been used to analyze or account for 
differences observed between native and non-native speech act utterancr s. Speech acts that have 
been studied to compare native and non-native pragmatic competence include expressions of 
gratitude (Eisenstein & Bodman, 1986,) apologies (Borkin & Reinhart, 1978; Cohen & Olshtain, 
1981), request (Blum-Kulka, 1982), complaints (DeCapua, 1989; House & Kasper, 1981), 
compliments (Wolfson, 1983a, 1989b), suggestions (Bardovi-Harlig & Hartford, 1990,) and 
refusals (Beebe & Cummings, 1985; Takahashi & Beebe, 1987; Beebe et al., 1990; Bardovi- 
Harlig & Hartford, 1991; Hartford & Bardovi-Harlig, 1992b). 

Among these speech acts, refusals are considered to be a face threatening act (FT A) in that 
either the speaker’s or the listener’s positive or negative face is risked when a refusal is called for 
or carried out. Consequently, refusals, as a sensitive, subtle, and high-risk FTA, can provide 
much insight into one’s pragmatics since the high level of pragmatic competence required in 
refusals distinguishes native and non-native pragmatic competence. 

2.3 Data Gathering Methods 

Pragmatic data can be gathered in many different methods, each of which has different 
limitations. In general, data can be collected by observation of real-life events, role-plays, written 
questionnaires (or Discourse Completion Tasks,) or metapragmatic judgement tasks. 

2.3.1 Authentic Data 
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Since pragmatics is highly context-driven, it is best illustrated when a speech act actually 
occurs and is in^ediately recorded as natural, authentic data, which document specific utterances 
^ (including ^^^s, tones, paces, pauses, etc.) with detailed description of the situation, the event, 
^^^i^^n^erbal reaction, status, gender, age, and relationships of the inierlocutors. The 
greatest strength of authendc data is that they have high internal validity as they are events that 
actually happened and are described in detail objectively. They also provide a rich context so that a 
researcher or reader, when judging pragmatic appropriateness of the interlocutors, can reconstruct 
the entire speech act event based on the natural data and the likelihood of his or her misinterpreting 
the situation is thus greatly reduced. 

However, natural, authentic data are very hard to come by as speech act events occur 
unpredictably and hence they are rarely recorded. In addition, it is highly unlikely that, in real-life 
situations, a given speech act occurs with the same event, in the same context, and/or with 
interlocutors’ of the same relationships. Because the variables in real-life speech act events not 
only are veiy complicated but also can hai’dly be held constant in order to make cross-event 
comparisons, speech act events can only be studied and analyzed as individual cases; generating 
abstract principle out of limited, individual, different situations can be difficult due to the frequent 
lack of a common base of comparison and therefore authentic data tend not to liave high external 
validity. 



2.3.2 Role Play 

Since authentic data are hard to come by, role play is sometimes used to collect naturalistic 
data by having subjects act out plausible pragmatic interactions under a given situation. Its 
advantage is that the pragmatic interactions observed in role play are contextualized; since role play 
often takes several turns in the discourse, it provide much insight into the meaning negotiation 
process. 

However, one primary drawback of role play as a data collecting method is that subjects 
tend to be obliged to produce the item the investigator is interested in studying (Larsen-Freeman 
and Long, 1991, p. 27) and thus result in Hawthorne effect. In addition, in doing role play, 
subjects understand that the situation is not a real-life one and there is no risk of threatening either 
the speaker’s or the listener’s positive or negative face. Consequently, they may, to a certain 
extent, exaggerate the pragmatic interaction just to create a role-play, dramatic effect and produce a 
pragmatic interaction substantially different from that in a real-life situation. 

2.3.3 Discourse Completion Tasks 

In a Discourse Completion Task (DCT), scenarios that call for specific speech acts are 
presented to subjects in written form; subjects respond in writing what they think they would 
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say under the situations described in the scenarios when ’iiey • re to carry out the speech 



act. Devices su^ as semantic taxonomy (Beebe et al., 1990) are then used to analyze DCT data by 



categorizing 



xises into semantic units to discover and compare native and non-native speecli 
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DCT enables the collection of a large amount of data so that bias can be reduced. In 
addition, because all subjects are presented with the same scenarios and respond in written form, 
data analysis tends to be more consistent and reliable. Beebe and Cummings ( 1985) have 
demonsttated that, for refusals, subjects’ intuitions about what they would say coiTespond closely 
to what other subjects actually did say in the same situation. However, because real-life verbal 
interactions involve much more elaboration, especially in face-threatening situations, using DCT as 
a tool to investigate speech acts has been somewhat controversial. Wolfson, Marmor, and Jones 
(1989) maintain that it is not reasonable to always assume that written responses are representative 
of spoken ones and that short, decontextualized written segments are not comparable to authentic, 
longer routines (p. 182). On the other hand, Beebe and Cummings ( 1985) claim that written 
responses are stream-lined, containing fillers for slots that the subject feels must necessarily be 
filled to serve a particular function (p. 201) and therefore adequately capturing the essence of their 
oral counterparts. In spite of the controversy, DCT is still the most frequently used method in the 
field, as authentic data of a specific speech act is hard to collect and compare and role play does not 
result in comparable data as DCT does. 

2.3.4 Metapragroatic Judgement Task 

Few studies have employed metapragmatic judgement tasks, possibly because of its 
difference from other data gathering methods that there has to be existing speech act data to enable 
the judgement task. In other words, a metapragmatic judgement task does not generate speech act 
'^ata but assesses them. It can be used as an interpretative tool with the aforementioned data 
gathering methods. In this study, a metapragmatic judgement task is employed. The assumptions 
are that 1) pragmatic impression is intuition-driven and the pragmatic appropriateness of a given 
statement should be intuitively and holistically judged by the speaker’s of that language, and 2) 
pragmatics is context- and discourse-based and a speech act statement is often more than the total 
of the semantic units that it is composed of; breaking down a speech act statement for analysis may 
alter or distort its pragmatic nature. 

The metapragmatic judgement task is employed in this study because it lends itself to the 
tliree research questions of this study: 



1 ) Whether what one considers pragmatically appropriate is also considered by other 
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speakers of the same language to be equally pragmatically appropriate, 

2) >Mj^her one’ s pragmatic judgement is consistent over time, and 

Whether statements made by native speakers tend to be considered pragmatically more 
" appropriate than statements by non-native speakers as judged by native raters. 



Apparently, the greatest strength of a metapnigr.iatic judgement task seems to be that it 
appeals to the subjects’ (i.e. raters’) intuition, which is the foundation of pragmatics and thus has 
high validity. That pragmatic intuition is holistic and subjective also makes a metapragmatic 
judgement task a reliable measurement instrument for the construct. In addition, it allows tlie 
comparison of intra- and inter-rater pragmatic judgement consistency and thus enables the 
investigation of whether pragmatic appropriateness of a statement as judged by a rater is consistent 
over time and also conforms to that by other raters. High consistency of intra- and inter-rater 
pragmatic judgement can in turn serve as a reliable indicator for the level of pragmatic 
appropriateness of individual speech act statements and thus allows the comparison of native and 
non-native pragmatic appropriatene.ss. 



3. METHODS 

3.1 Subjects 

Forty-two {20 male and 22 female) native English-speaking college undergraduate students 
participated in this study. These subjects are of similar age rank and share common cultural 
experience. Because the study was conducted in two different times, the subjects, according to 
their schedule, into two groups — Group A (13 subjects) and Group B (39 subjects). 

3.2 Instrument — Questionnaires 

Three questionnaires (i.e. I, II, & III) were used in this study. The questionnaires were 
based on a study conducted earlier (Chen, 1991 ), in which four scenarios that call for the speech 
act of refusals were given to 26 native and non-native speakers of English. Respondents were 
graduate students of different majors in Indiana University. The non-native subjects included 7 
Chinese, 5 Japanese, 4 Koreans, and S Europeans, and there were 5 native subjects. These 
subjects responded to a DCT in which there were eight scenarios representing four types of 
refusal-eliciting stimuli (i.e. requests, invitations, offers, and suggestions (adopted from Beebe et 
al., 1990).) Cultural bias was minimized by carefully selecting and wording the scenarios. These 
scenarios and the subjects’ responses in tlie DCT were randomly selected to compose 
Questionnaire I (see Appendix A) of this study. 

In Questionnaire I, there were four scenarios with six responses for each. Of the six 
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responses, two were made by native speakers and four by non-native speakers of English in the 
1991 study. It^nationalities and the language background of the respondents providing the 
statements )^^liot available on the questionnaires. Neither was there any indicator as to whether 
were made by native or ncm-native speakers. A five-point Likeit scale was given to 
the subjects in this study to rate the appropriateness of each speech act statement (i.e. response) in 
the scenarios. The rating ranged from “very inappropriate” as “1” to “very appropriate” as “5” on 
the scale. 

The subjects were Instructed to take an “outsider” perspective (as opposed to an 
interlocutor’s perspective) and rate the appropriateness of each statement under the situations 
described in the scenarios. Specifically, the subjects were told that '"Imagine that you happen to 
eavesdrop or witness a situation as any of those scenarios on the questionnaire, and someone 
responds with something as any of those responses under the scenarios. As an outsider, what 
would you think the appropriateness of each of the statement is under the situations?” The subjects 
were given an outsider’s perspective in order for them to give subjective ratings without being 
involved in the situations so as to avoid any bias effect. If the subjects were involved in the 
scenarios as the interlocutors, they might translate the scale into “how likely or often would I make 
such a statement under the situation.” In this case, feeling that they might be judged personally by 
the ratings they gave, they would want to project a best possible self-image by not giving their 
authentic opinions. 

Four weeks later, the subjects were given a post-test with the same instruction. Some 
subjects recognized inunediately that the scenarios and statements in the posttest were the same as 
those in tlie pretest but were still instructed to read through the scenarios and truthfully rate the 
statements according to their opinions. 

In the posttest, subjects in Group A were given Questionnaire II and subjects in Group B 
Questionnaire III. (See Appendix B for the illustration of the research design.) Questionnaire II 
and ni are composed of the same scenarios and speech act statements as those in Questionnaire I 
but with different scenario or statement orders. In Questionnaire II, the internal order, or order of 
speech act statements of Scenario II and IV, was randomized within the two scenarios. In 
Questionnaire IK, the external order, or order of the four scenarios, was randomized. The purpose 
of randomizing the statement- and scenario-order in Questionnaire II and III was to investigate 
whether there was an order effect (i.e. the order in which the subjects respond to the statement or 
scenario effects the ratings.) 

3.3 Data Analysis and Discussion 

All of the subjects’ pretest and posttest data were entered and pre-post differences were 
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calculated (see Appendix C). The randomized orders in Questionnaire II and HI were matched 
with the scena^fc^d statement-orders in Questionnaire I. In the following discussion, the 
i^raement numbers refer to those in Questionnaire I. For instance, IV- 1 refers to 
V, Statement (a) in Questionnaire I. 

There were five foci in analyzing the pretest and posttest data: 




1 ) Frequency distribution of each statement under each scenario 

2) Statement means and standard deviations 

3) Consistency of pre-and-post ratings for individual statements across 
subjects 

4) Consistency of pre-and-post judgement by individual subjects across 
statements 

5) Order effect 



3.3.1 Frequency Distribution 

Frequency distribution was calculated for each statement (see Appendix D.) Two 
distribution patterns emerged for the 24 statements of the four scenarios: 

a. Clustered Distribution'. 

The frequency distribution tended to cluster or peak towards either the higher or the 
lower end of the Likert scale for the nnajority of the statements. These statements include 1- 1 , 
1-2, 1-3*, 1-4, T-6*. II- 1*. n-2, n-4*, H-S, II-6, III-l, III-2, ni-3*. III-4, ni-5*, III-6, IV- 
1, IV -2*, IV -3, IV-4*, & IV -6 (* denotes statements made by native speakers of English in 
the 1991 study). 

Depending on which end of the scale they cluster towards, some of these statements 
can be furthered categorized into two types: high-appropriateness cluster and low- 
appropriateness cluster. All of the statements made by native speakers of English show a 
high-appropriateness cluster, whereas statements with a low-^propriateness cluster were 
made by non-native speakers of English (including 1-2, 1-4, 11-2, II-5, II-6, III-4, and 0-6.) 

b. Scattered Distribution: 

There are a few statements that either do not have as sharp a peak or lack a clear 
tendency on the distribution. These statements are 1-5, II-3, and IV-5. 

3.3.2 Means and Standard Deviations 

Appendix E shows the mean and the standard deviation of each statement. As 
aforementioned, within each scenario, almost all the statements made by native speakers have 
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liigher means of appropriateness in comparison with statements made by non-native 
cnAntorc.- (^examining all the native statements, it seems that what makes these statements 



opriate are that 1) Native statements attribute the reasons of their refusals to some 
'Vernal, pre-existing, impersonal, or uncontrollable situations, such as “I am 
going to a concert and we bought the ticket a long time ago" (II-4*) in turning down a dinner 
invitation, and ‘‘I need the car/notes myself’ (1-3*, IV-2*, & IV4*) in turning down a 
request for borrowing the car/notes. By externalizing or impersonalizing the reasons of 
refusals, the speaker avoids threatening his or her positive face (i.e. one’s desire of being 
liked) and/or the listener’s negative face (i.e. one’s desire of not being imposed upon); 2) 
Native statements elaborate the reasons of refusals. Perhaps attempting to reduce the 
intensity of the situation as caused by the refusals and to support the reasons, native refusals 
in general are lengthier and better elaborated than non-native refusals; 3) Native statements 
offer (well-intended) altertiatives or suggestions and thus open up other doors for the 
listener, instead of abruptly stating “no”. Some alternatives or suggestions are “(I can’t lend 
you all my notes, but) is there one particular class that you need some notes on?” (1-6*) 
“Maybe someone in your study group can loan you the notes,” (1-3*) “...But, could we go to 
lunch Tuesday together?” (II-l *) and “Perhaps we can get together some other time” (U-4*); 
and 4) Native statements down grade the refusals, by minimizing the undesirable 
outcome that the listener caused and offers to make up for. For instance, “Don’t v/orry. 
Really...'” “...It’s nothing at all. I’ll gc ' 'it in a minute.” (IU-3*) as refusals to the listener’s 
offer to help cleaning up the spilt coffee and thus make the listener feel “off the hook.” 

In contrast, standard deviation is low for several non-native statements that also have 
low appropriateness average, suggesting that these non-native statements are very 
consistently considered by most raters to be very inappropriate. Specifically, these non- 
native statements are “I think you should think about what you are doing” (1-2) as a refusal to 
lend out notes, “Is your husband/wife going to be there? If so, no. Thanks.” (II-5) and 
“Why don’t you get a divorce?” (II-6) as refusals to a friend’s invitation to dinner, and “Let 
me do it. You’ll only make matters worse” (II1-4) as a refusal to ttie offer of helping to clean 
up. What seems to be common about these highly inappropriate statements is that imbedded 
in the refusals are additional messages that criticize the listener personally, with an overtone 
suggesting the reasons of refusals being something negative about the listener, similar to 
“you should be ashamed that you not only didn’t come to class but also want someone else’s 
notes that you didn’t work for,” “You are married to a wrong person,” and “You are so 
clumsy that I can’t let you help me clean up the mess you made.” It is difficult enough to 
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preserve the speaker’s positive face when carrying out a refusal; the speakers in the above 
situations^e even one step further to threaten the listeners’ negative face by making 
persQ^^Icnticisms. Failure to maintain the listener’s negative face in the native way is 
^^cwse of the pragmatic inappropriateness in these non-native statements. Native 
strategies to save the listener’s negative face include externalizing / impersonalizing (as 
opposed to internalizing / personalizing) the situation and minimizing the undesirable 
outcome caused by the listener (as opposed to emphasizing the possible negative 
consequence). The best example to illustrate this point is “Let me do it. You 'll only make 
matters worse .” in the spilt coffee scenario (IH-4, mean=l .64.) The speaker of this 
statement personalizes the refusal (cf. “This kind of things happen. Things break,” (III-2, 
mean=4.01) which externalizes and impersonalizes the situation.) In addition, the speaker of 
ni-4 also emphasizes the negative consequence (c,f. “It’s nothing at all...” (111-3*, 
mean=4.0l)) and appears to have failed to save the listener’s negative face. 

Also, a closer examination of the aforementioned four highly inappropriate non-native 
statements (i.e. 1-2, II-5, II -6, and ni-4) suggests that not only do they fail to employ (in the 
native way) the appropriate refusal strategies atid to preserve the listener’s negative face, 
some of them further aggravate the inappropriateness by making personal criticisms under an 
inferior status of favor. The violation of status of favor is another cause of pragmatic 
failure for some of these highly inappropriate statements. These four statements illustrate 
two situations: refusals to someone’s request (e.g., borrowing notes/car) and refusals to 
someone’s offer (e.g., to dinner or to help cleaning up). These two situations differ in that, 
in the first situation, the person making refusals is a possible favor-granter and thus is in an 
advantaged or favor-superior position, while in the second situation, the person making the 
refusal is a possible favor-receiver and thus is in an disadvantaged or favor-inferior position. 
It is already presumptuous for someone in an inferior, favor-receiving position to refuse, and 
thus threaten the positive face of the superior favor-granter; the inappropriateness is even 
further aggravated by the favor-receiver turning down the well-meaning favor-granter with 
personal criticisms or attacks (e.g., suggesting a divorce, which differs very much from the 
well-intended altemative/suggestion strategy observed in the appropriate, native statements). 
The observation that the status of favor is a factor of pragmatic appropriateness is supported 
by the fact that the refusals made by the inferior-status-of-favor threatening the negative face 
of the superior-status-of-favor (i.e. II-5, II-6 and III-4; refusals with personal criticisms to 
dinner invitation and to offer cleaning up) have the lowest appropriateness means (1 .61, 1.04 
and 1.64 respectively) of all statements. In comparison with these three refusals by inferior 
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status of favor, the refusals by superior status of favor (1-2), though highly inappropriate, 
still has a^gnificantly higher mean of appropriateness (2.71), because, although the 



listenefll^gative face is threatened by the speaker’s personal criticism, the speaker has more 



le right to do so as he/she is in a superior status of favor. 

A comparison of DI-5 and m-6 (both are refusals to offer help to clean up spilt 
coffee) renders insight into another possible cause of pragmatic inappropriateness. The two 
statements are similar in that they both externalize the reasons of refusals (“I have some 
handy cleaning stuff..." in III-5 and "I’m going to buy the cleaning kit later" in III-6.) 
However, one (1II-5) results in a high appropriateness mean (4.17) and the other (HI-6) a 
much lower mean (2.62). The difference seems to lie in the degree of semantic clarity of the 
reason of refusals, ni-5 expresses clearly that the speaker has a handy stuff for the problem 
and thus refuses the listener’s offer to help, while II1-6 does not make as clear a refusal (i.e., 
buying the cleaning kit later does not readily translate into an acceptance or a rejection of the 
listener’s offer to help.) This seems to suggest that, although not as detrimental, semantic 
ambiguity can be a cause of pragmatic inappropriateness. Another example of semantic 
ambiguity causing pragmatic inappropriateness is TV-5 (mean=2,63), “I might go 
somewhere. I’ll give you may answer later'' as a refusal to lending out the car (cf. semantic 
clarity in “I rieed it this week' in IV-2* and IV-4*, or even "I can 't lend you my car" in IV-6, 
all of which have a higher mean than that of the semantically ambiguous statement, IV-S). 

3.3.3 Statement Rating Consistency 

Level of consistency for the rating of each statement across raters is also calculated. 
The Statement Rating Consistency table in Appendix F shows that, on average, more than 
half (i.e. 55.3%) of the posttest statement ratings are identical with their corresponding 
pretest ratings (i.e, ±0, or differ by 0 point on the five-point scale), a considerable percentage 
(i.e. 35.5%) of the posttest statement ratings differ from their corresponding pretest ratings 
by only 1 point (i.e. ±1) on the five-point scale, and only 8% of the posttest ratings differ by 
2 points (i.e. ±2) or more on the five-point scale from their corresponding pretest ratings. In 
other words, the statements, on average, have a very high pretest-posttest rating consistency 
across subjects, which, in turn, suggests a very high inter-rater reliability. 

Examination of individual statements on the table shows that several of them have 
eve higher pretest-posttest consistency (II-6 and III-3*, with 93% and 74% respectively for 
±0 difference). Also, at the ±0 level, there are three statements that have significantly lower- 
than-average percentages. These are 1-4, 1-5, and II-3, with 33%, 40% and 40% 
respectively at the ±D level. However, these three statements “catch up" on their level of 



O 

ERIC 



14 



Metapragmatio Judgement on Refusals 

14 




fflftciitlfiitBkiffi 



pretest-posttest rating cx>nsistency with significantly higher-than-average percentages (48%, 
% respectively) at the ±1 level, suggesting tliat, although these tliree statements 



ly lower percentages of identical pre-post ratings, they are not “far off* as over 
itings in each of these three statements differ by 1 point or less in the posttest. 

On the other hand, investigating the pragmatic nature of tliose statements that have 
significantly higher or lower percentages of pre-post rating consistency may provide us with 
some insight into what influences pre-post rating consistency. It seems the three statements 
(1-4, 1-5. and II-3) that have lower-than-average pre-post rating consistency at the ±0 level 
have in common that they are semantically vague and can be pragmatically translated in many 
ways and thus result in lower pre-post rating consistency. The statement that **If you had 
been in class, they (the notes) would make sense to you” (1-5) does not really tell the listener 
whether he/she can borrow the notes. The lack of semantic clarity in this statement grants the 
rater greater latitude in interpreting its pragmatic message and this may be the reason why this 
statement also has a scattered distribution and a high standard deviation (1.11). The 
statement that “You should’ve taken notes by yourself!” (1-4) is similar to 1-5 in the same 
way. This observation that semantic clarity influences the consistency of pragmatic rating is 
in line with the aforementioned distinction Leach (1983) draws between “sentence meaning” 
and “speaker’s meaning;” the incongruity of these two meanings, caused by the lack of 
clarity in “sentence meaning” in the above examples, results in a higher latitude for pragmatic 
interpretation and thus a lower consistency in pre-post rating. The mismatch between the two 
types of meanings also conforms with what Thomas (1983) calls “pragmatic failuie” and 
explains why these statements have somewhat lower means (2.64 for 1-4 and 2.94 for 1-5) of 
pragmatic appropriateness. 

II-3 is the third statement that has a lower-than-average pre-post consistency 
percentage at the ±0 level. This statement seems to project a different reason for lower pre- 
post consistency. The reason, instead of semantic ambiguity, is an opposite one in that it 
states the truth (that the speaker does not like the spouse of the listener) in an honest way, 
assuming that the listener already knows the reason (“You know that I don’t get along with 
your husband/wife”) and straightforwardly suggests the possible negative consequence 
of the event (“It’ll be awkward when we are together”) as the reason for the refusal. 
Apparently, there is a high congruence between the sentence meaning and the speaker’s 
meaning in this statement. What causes it to have a scattered distribution, a high standai'd 
deviation, and a relatively low pre-post rating consistency may be the different pragmatic 
strategies (truth, honesty, expressing negative consequence) that can be highly individual- 
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case-based and that people in general do not have a strong preference for or prejudice against. 
Thu :r these strategies are positive or negative by themselves and tvhether the 



mohusea oy lack of tactfulness and well-roundedness in the way the truth is told are very 
much left to a rater’s personal judgement. Given the many complicated, and even 
contradictory qualities of this statement, it is likely that the rater may have a mixed pragmatic 
feeling about the statement and therefore gives it a considerably different rating of pragmatic 
appropriateness in the posttest, resulting in the statement’s lower pre-post rating consistency 
at the ±0 level. 

Although not a statement of low pre-post rating consistency at the ±0 level, statement 
rV-3 forms an interesting comparison with II-3 and hence helps to more closely examine the 
interaction of the many qualities in II-3. IV-3 is similar to 11-3 in that it also expresses a 
negative consequence as the reason for refusing to lend out the cai' (“If something happens, 
it’s a mess with the insurance.’’) Whereas both 11-3 and IV-3 have a mean indicating positive 
pragmatics (3. 1 for H-3 and 3.75 for IV-3), IV-3 is undoubtedly pragmatically more 
appropriate than II-3. This difference may be attributed to the fact that I'.'^ 3 expresses the 
negative consequence (that the insurance can be messed up) as the reason for refusal without 
alluding to the truth (that the listener is a bad driver,) which would threaten th", listener’s 
negative face. On the other hand, in II-3, although telling the truth (that the speaker does not 
get along with the friend’s spouse) is a virtue and can be considered a positive strategy, it 
nonetheless violates the mle that the listener’s negative face should be saved. 

In addition, all these three statements (1-4, 1-5, and II-3) that have a lower-than- 
average pre-post rating consistency at the ±0 level also have means very close to the middle 
of the five-point scale (i.e. “3”.) The middle (or “mild”) ratings indicate that raters either did 
not have a strong opinion or have varied opinions about these statements. 

As for the statements (II-6 and III-3) that have extremely high pre-post rating 
consistency at the ±0 level, although one pragmatically negative (mean=1.03 for II-6) and tlie 
other positive (merui=4.01 for III-3), they have in common the clear tendency of high 
(in)appropriateness; their means fall at the ends of the five-point scale and are either highly 
pragmatically ^propriate or extremely pragmatically inappropriate. This tendency is also 
illustrated by their frequency distributions having sharp peeks at the higher/lower ends. 

This observation that highly (in)appropriate statements tend to have strong pre-post 
rating consistency (at the ±0 level), in conjunction with the previous discussion about 
semantically vague or pragmatically contradictoiy/compEcated statements tending to have a 



posi 




le of truthfulness and honesty in the statement outweighs the negative impression 
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mddle/mild pragmatic rating and thus a lower pre-post rating consistency (at the ±0 level), 
indicates |l^at there does exist a correlation between the level of pre-post rating consistency (at 
thed and the “strength” of the positive or negative pragmatic feelings a statement 

[ere the raters. A rater, when giving a very high or low pragmatic rating to a statement. 



tends to have a strong opinion about the pragmatics of the statement, and, since strong 
opinions tend to stay stable, the rating the rater gives to that statement in the posttest tends to 
have high consistency with that in the pretest. This correlation is quite obvious as illustrated 
in Figure 1 below: 



Insert Figure 1 about here 



3.3.4 Subject Judgement Consistency 

Appendix G shows the table of each of the 42 subjects’ pretest-posttest Judgement 
consistency across statements as projected by their pragmatic ratings. On average, 52.6% of 
the subjects’ pragmatic judgement in the posttest matches identically (i.e. ±0) with their 
pragmatic judgement in the pretest. A considerable percentage (34.9%) of the subjects’ 
posttest pragmatic judgement differs from their pretest pragmatic judgement by only 1 point 
(i.e. +1) on the five point scale. Less then 13% of the subjects’ posttest pragmatic 
judgements differ from their pretest pragmatic judgements by 2 points (i.e. ±2) or more on 
the five-point scale, and 2.5% differ by of 3 points (i.e. ±3) or more. There are a couple of 
subjects that have relatively low pre-post pragmatic judgement consistency at the ±0 level 
(33% for Subject B8 and 29% for Subject B21). However, their judgement consistency at 
the±l level also indicates that they “catch up” on their level of judgement consistency with a 
significantly higher-than-average percentage (50% for Subject B8 and 58% for Subject B21 ), 
so that, combining both the ±0 and the±l levels, they still manage to maintain a percentage 
higher than 80% of ±0 or ±1 pre-post judgement consistency, and this percentage is by no 
means significantly different from the average (i.e. 87.5% as the total of the ±0 and ±1 
levels). 

The subjects’ high consistency of pretest-posttest pragmatic judgement indicates a 
high intra-rater reliability. 

3.3.5 Order Effect 

It might be possible that the first scenario/statement that the subjects responded to 
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^anchored” the ratings for the ensuing scenarios/statements, which would be compared to 
and influ^ced by how high or low the rating of the first scenario/statement is. It might also 
be p 9 |^^^at, since the scenarios and statements in the posttest are identical with those in 
■^4&tSest, the subjects might have recalled during the posttest the ratings they gave in the 
pretest not (only) because of the actual pragmatic qualities of the statements but the order in 
which the scenarios/statements were presented to them in the pretest. Consequently, order 
effect is examined by comparing the three questionnaires (which only differ in their scenario 
or statement order). The Order Effect table in Appendix H shows that, in comparing 
Questionnmre I with Questionnaire II (which differ in the intemal/statement order within 
Scenario II and IV,) Scenarios I and HI, whose statement orders remain the same in the two 
questionnaires, do not yield a total of rating difference significantly different from those of 
Scenario n and IV, whose statement orders are randomized in Questionnaire 11. Becau.se of 
the lack of significant difference between the randomized and the unchanged statement 
orders, it seems that there is no order effect in the questionnaires and that the ratings the 
subjects gave to the statements are independent from the order in which the statements were 
presented to the subjects. 

In addition, because of the absence of order effect, as observed among subjects in 
Group A (who had Questionnaire n in posttest), it is assumed that there is also no order 
effect in Questionnaire IH, which were given to subjects in Group B in the posttest with the 
scenario order randomized. 



4. RESULTS 

The above observations from the data help answer the three research questions projected 
earlier. The questions are: 

1) Is what one considers pragmatically appropriate also considered by other 
speakers of the same language to be equally pragmatically appropriate? 

The answer seems to be yes. The strongest evidence is that there is a very high 
statement rating consistency; in addition, the majority of the statements show clustered 
distributions, suggesting that the raters tended to have similar, if not identical, opinions 
regarding the pragmatic appropriateness of the statements. 

The data analysis also suggests what the raters consider to be pragmatically 
appropriate and what inappropriate. The pragmatic strategies that are found among 
statements rated pragmatically appropriate include: 

• Externalizing or impersonalizing the situation of refusals, 
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Elaborating the reasons of refusals, 
i^^^Mering alternatives or suggestions, and 
T^inimizing the undesirable outcome the listener caused. 
pflgmatic strategies that are found among statements rated pragmatically inappropriate 
include: 



• Failure to preserve the listener ‘s negative face, 

• Making personal criticisms, especially under an inferior status of favor, and 

• Semantic ambiguity. 



In addition, although the statement rating consistency is very high, the raters seemed 
to agree more strongly on some statements than on others. The statements that the raters 
showed strong rating agreement have the following characteristics: 

• They have a clear tendency to have extreme scores, either highly expropriate or 
highly inappropriate, and 

• Their frequency distributions tend to cluster and have sharp peeks. 

The statements that the raters showed less strong rating agreement have the following 
characteristics: 

• Their scores tend to fall in the middle oj the scale, 

• They are of complicated or contradictory pragmatic qualities, and 

• They may be semantically vague of pragmatically mild and thus they do not cast a 
strongly positive or negative impression on the raters. 

The rating consistency in combination with the means of the statements’ pragmatic 
appropriateness also suggests a correlation between the strength of the positive or negative 
impression the statements render the raters and the consistency of the rating the statements 
have; the stronger the pragmatic impression, the more extreme the ratings, and the higher the 
level of rating consistency. 

2) Is one’s pragmatic judgement consistent over time? 

The answer for this question also seems to be yes. As discussed, the subjects’ 
pretest-posttest judgement consistency is very high, implying that the raters tended to hold a 
stable pragmatic opinion over time for a given statement 

3) Do statements made by native speakers tend to be considered pragmatically 
more appropriate than statements by non-native speakers as judged by native 
raters? 
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Once again, the answer to this question seems to be yes, which is supported by 1) 
common ?fiptegies or qualities are found among the native statements but among only very 
few n^^ktive statements, supporting that people speaking the same language share common 
Ts^s of expressing refusals, and 2) the raters in this study very consistently rated these 
native statements as highly pragmatically appropriate, indicating that people speaking the 
same language tend to agree on and approve of the ways in which other speakers of the same 
language commonly express refusals. 



5. LIMITATIONS 

Like most research on pragmatics, this study’s limitations are due to its sample, tlie data 
collection method, and data analysis and interpretation. 

5.1 Subjects 

Since the 42 subjects participated in this study were college undergraduates with the 
same age range, homogeneous linguistic experience, and similar cultural background, they 
may very well under-represent the entire language population. However, it is difficult to 
determine who the representatives of a language population should be. Do they include 
people who have rich cross-cultural, multi-lingual- experiences or even have lived m another 
culture and speak another language? Culturally and linguistically experienced people may be 
disqualified for the purpose of investigating the sociocultural pragmatic pattern of the 
language, as they might already be “contaminated" by the culture or language they have 
experience in. If the ideal subjects should be culturally and linguistically homogeneous, the 
subjects in this study then is quite representative. On the other hand, even among the 
culturally and Linguistically homogeneous population, there are different variables that can 
influence one’s pragmatic opinion. These variables may be one’s age, up-bringing, gender, 
personality, sociocultural exposure, etc. The subjects of this study, in this sense, may fail to 
include many of the different variables. 

5.2 Data Collection 

Since the questionnaires this study used were based on a study conducted earlier 
(Chen, 1991), in which DCT was the format for collecting written responses, the 
aforementioned potential drawbacks of DCT apply to the statements in the questionnaires of 
this study. However, in spite of this limit, the statements in the DCT were transformed into 
the questionnaires without threatening their validity or distorting the data, since both the 
subjects in the 1991 study who produced the statements and the subjects in this study who 
rated the statements responded to/in the same written form. 
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In addition, although four weeks is a reasonable amount of pretest-posttest interval, it 
is hard to Jt|dge whether that is a time span long enough for the subjects to foiget the 



numb^p^y associated the individual statements with in the pretest in order to react tc the 
^l^ments purely intuitively and pragmatically in the posttest. 

5,3 Analysis and Interpretation 

Because the interpretation of the data is based on the limited number of the statements 
(total 24 statements for four scenarios.) the support and generalization of the findings are 
limited. However, there is also a limit as to how many statements the raters may have 
patience and attention for. Another issue is the insufficient triangulation for the validity of the 
findings and the interpretation. Although some tiiangulation was found among the 
statements regarding what seemed to be the characteristics of appropriate pragmatics and 
what inappropriate, another person investigating the same data might have a different view, 
suggesting the next phase of the study to be conducting interviews to gather opinions about 
the pragmatic appropriateness of the statements or having other investigator interpret the 
data. 



Yet another issue concerns whether the refusals in the statements are true reasons or 
false excuses. Prior to this study, the questionnaires were tested and given to several people 
to react to. One of the people gave double ratings for several statements and ir licated that 
one rating was for “if true” and the other “if false,” with the two ratings significantly different 
fre m each other. It seemed that, at least to this person, whetJier a statement was actually tme 
or false (i.e. whether the speaker of the statement was telling the truth or lying, in whatever 
pragmatic way) was a factor of its pragmatic appropriateness. This “true or false" issue is a 
problem not as much of the questionnaires or statements but of the construct of pragmatics. 
Even in real-life situations, rarely can we find out whether one, in saying no, is telling a tme 
reason or a false excuse. Consequently, this “tru. -)r false” issue calls for a specific 
definition of pragmatics and a clear distinction between pragmatics and intention. As 
aforementioned, a meaning can be twofold, sentence meaning (i.e. semantic meaning) and 
speaker’s meaning (i.e. pragmatic meaning) (Leech, 1983). When the two meanings of an 
utterance mismatch without the speaker’s knowing it (i.e. the speaker means one thinj and 
does not know he/she is heard to have said another), pragmatic failure (Thomas, 1983) 
occurs; however, when the speaker means one thing but intentionally says another, the issue 
is not of pragmatics. Going back to the “true or false” issue, the responder ts’ not knowing 
whether the speakers of the statements had a pragmatic failure or actually intended to give a 
false excuse assimilates real-life situations very well. As we form impressions about people 
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often without knowing or even tliinking whether something they say is of pragmatics or 
'ntention,^ subjects in this study were only instructed to pretend that they had heard the 
stateipj^p^ey rated without the investigator’s alluding to or sensitizing them with the “true 



6. CONCLUSION 

The flndings in this study, although they distinguish native and non-native pragmatics and 
thus suggest the language- and culture-specificity of pragmatics, do not by themselves generate a 
pattern. Given the subtlety and complexity of the construct, pragmatics needs to be investigated 
from different perspectives and in different formats. Individual pragmatic cases encountered in real 
life situations provide rich insights all by themselves, discourse completion tasks collect large 
amount of data, questionnaire surveys yield metapragmatic data that are consistent in form and also 
enable comparison and generalization, selective rater interviews further explore the raters’ opinions 
and possibly how one’s experience influences his/her pragmatic Judgement, and inter-investigator 
triangulation validates findings from, and inteipretations of, the data collected in the above various 
formats. Only in so doing can the nature of pragmatics be thoroughly studied and the many cross- 
cultural, inter-language pragmatic questions be appropriately discussed. 
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Appendix A 
Questionnaire 1. 

Given the following four situations (I ~ IV), how appropriate do you consider each 
^ of their response |A ~ F) to be? Give each response in each situation a rating, by 



l^cling one oj 



five numbers on the scale besides it. 





2 


3 4 


5 


very inappropriate 


inappropriate 


undecided appropriate 


very appropriate 



I. W attends classes regularly and takes good notes. One person in W’s class who doesn't show up very 
often asks to borrow W’s notes. Since W has to compete with the rest of the class to earn a good grade, 
W doesn 'ifeel like sharing the results of his/her hard work with someone who doesn 't work for it. W 
says. 



1 2 3 i 5. A) “If I lend my notes to you, it is unfair to me and others who come to class 

regularly.” 

± 2. 3. 4 5. B) “I think you should think about what you are doing,” 

1 2. 3. 4 5. C) “I need them to study from. Maybe someone in your study group can loan 

you the notes. Sorry.” 



1. 2. 3. A S. D) “You should’ve taken notes by yourself.” 



1 2. 3. A S. 



E) “If you had been in class, they (the notes) 'vould make sense to you.” 



1 2. 3. A. S. P) “I’m not sure my notes will help you because they relate so closely to what 
was said or done in class. I really would rather not have them all copied. Is 
there one particular class that you need some notes on?” 



//. A friend invites X to dinner, but X really can 't stand this friend’s husband/wife. X says. 



1 2. 3. A S. 

1 2. 3. A S. 

1 2. 3. A 5. 

1 2 3. A S. 

1 2 3. 4 5. 

i 2 3. 4. 5. 



A) “Well, I can’t that night. But, could we go to lunch Tuesday together?” 

B) “I don’t want to go out at night.” 

C) “You know I don’t get along with your husband/wife. It'll be awkward when 
we are together.” 

D) “I’m sorry, but my husband and I are going to a concert. We bought the 
tickets a long time ago. Perhaps we can get together some other time.” 

E) “Is your husband/wife going to be there? If so, no. Thanks.” 

F) “Why don’t you get a divorce?” 






ERIC 



I -p.i 




^ (T%ii Y*s friend is^ A house for coffee, 
^’s makes a mess of it. 

‘^€^i»^rimd to. Y says. 



The friend accidentally spills a whole cup of coffee on 
The friend insists on cleaning it up for Y, but Y doesn't 



1 2. i i 5. 

1 2. 2. 1 S. 

1 2. 3. 4. 5. 

1 2. 3. 4. 5. 

1 i 3. 4. S. 

12 14 5 . 



A) “It’s my house. I don’t want my guests to clean up anything.” 

B) “This kind of things happen. Things break.” 

C) “Don’t worry. Really. It’s nothing at all. I’ll get it in a minute.” 

D) “Let me do it. You’ll only make matters worse.” 

B) "No, no. Don’t worry, I have some handy cleaning stuff 1 use for stuff like 
that. Sit still and keep talking. I’ll get it.” 

F) “I’m going to buy the cleaning kit later.” 



IV. Z's roommate asks to use Z's car to go to Chicago. Knowing that the roommate is a careless and 
unskillful driver, Z doesn 't want to lend the roommate the car. Z says. 



1 2. 3. 4. S. 

1 2. 1 4 5. 

1 2. 3. 4. S. 

L 2- 1 4. 5. 

1 2. 1 4 5. 

1 2. 3. 4. S. 



A) “I’m sorry, but I don’t lend my car to anyone.” 

B) “I’m sorry, but I don't feel comfortable loaning out my car for long trips. 

Besides, I need it this week. How would I get around?” 

C) “If something happens, it’s a mess with the insurance.” 

D) “I’m sorry, but 1 made a policy not to lend my car to anybody. Besides, I 
need it this week.” 

E) “I might go somewhere. I’ll give you my answer later.” 

F) “I’m sorry. I can’t lend you my car.” 



I - p.2 
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Appendix B 
Research Design 




n=13 n=29 



Pre-Test: Q'naire 1 - Q'naire 1 



1 

(4-week interval) 

4 



Post-Test: — -Q'naire 2 - Q'naire 3 

(Randomized order of (Randomized order 

Statements within of Scenarios) 

Scenarios li & IV) 



20 
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Pretest & Posttest Scores 
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Appendix C 
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Individual Subject Pretest-Posttest Differences 







Qr. 





Mean: Std. Dev.: 

3.69 .606 



Mean: Std. Dev.: 

2.714 .899 



Mean: Std. Dev.: 

4.071 .929 




1-4 



Mean; Std. Oev.: 
2.643 1.199 




1-6 

Mean: Std. Oev.; 

2.94 1.112 




1 2 3 4 5 

1 - 6 - 

Mean: Std. Oev.: 

3.667 .986 
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Statements made by native speakers of English 
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Count Count 



Scenario II. Frequency Distribution 




60 

45 

40 

35 

30 

§ 25 
o 
O 

20 
15 
10 
5' 

ojI 

1 2 3 4 5 

11-2 





n-3 



Mean: Std. Dev.: 

4.012 .885 



Mean: Std. Dev.: 

2*381 .877 



Mean: Std. Dev.: 

3.095 1 .35 






Mean: Std. Dev.: 

3.667 1.123 



Mean: Std. Dev.: 

1 .607 .878 



Mean: Std. Dev.: 

1.036 .187 
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Count Count 



Scenario III. Frequency Distribution 






Mean: Std. Dev.: 

3.452 1 .069 



Mean; Std. Dev.: 
4.012 .784 











ll!-3 * 



Mean: Std. Dev.; 

4.012 .829 




Mean: Std. Dev.: 

1.643 .755 



* Statements made by native speakers 




Mean: Std. Dev.: 

4.167 .789 



English 




Mean: Std. Dev.: 

2.619 .863 



O 
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Count Count 



Scenario IV. Frequency Distribution 








Mean: Std. Dev.: 

3.964 .813 



Mean: Std. Dev.: 

4.143 .794 



Mean: Std. Dev.: 

3.75 1.005 




IV-4’ 

Mean: Std. Dev.: 

4.036 .842 




IV-5 

Mean: Std. Dev.: 

2.631 1.128 




IV-6 



Mean: Std. Dev.: 

3.94 .998 
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Appendix E 

Statement Means and Standard Deviations 




1 Statements 


Mean 


Descending Rank 


SD 


Ascending Rank 


1-1 


3.69 


2 


0.81 


1 


1-2 


2.71 


5 


0.90 


2 


1-3* 


4.07 


1 


0.S3 


3 


1-4 


2.64 


6 


1.20 


6 


1-5 


2.94 


4 


1.11 


5 


i-6* 


3.67 


3 


0.99 


4 


ii-r 


4.01 


1 


0.89 


4 


11-2 


2.38 


4 


0.88 


2 


lf- 3 ' ’ 


3.10 


3 


1.35 


6 


11-4* 


3.67 


2 


1.12 


5 


il-5 


1.61 


5 


0.88 


3 


[i-6 


i .04 


6 


0.19 


1 


IIM 


3.45 


4 


1 .07 


•"6 


111-2 


4’. 01 i 


2/3 


0.78 


2 


Iti-3* 


4.01 _ 


2/3 


0.83 


4 


111:4 


1",64 


6 


0.76 


1 


!i!-5* 


4.17 


1 


0.79 


3 


lil-6 


2.62 


5 


0.86 


5 


IV-1 


3.96 


3 


0.81 


2 


iV-2* 


4.14 


1 


0.79 


i 1 


lV-3 


3.75 


5 


1.01 


I 5 


iv-4* 


4.04 


2 


0.84 


i ^ 


IV-5 


2.63 


6 


1.13 


6 


tV-6 


3.94 


4 


1.00 


I 4 
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Appendix I 

Statement Rating Consistency 






State menls 


± 0 


± 1 


±2 


t- 


3 


± 4 




Total 


1-1 


2 1 1 50% 


18 


43% 


3 


7% 


. ..j 


_ 0®/o 




0% 




100% 


1-2 


23 


55% 


13 


31% 


5 


12% 


1 


”2% 




0% 




100% 


1-3* 


21 i 


50% 


18 


43% 


3 


7% 




0% 




0% 




100% 


1-4 


14 33% 


20 


48% 


4 


’16% 


4 


1 0% 




0% 




1 00% 


1-5 


17 


40% 


18 


43% 


5 


' 12% 


1 


2% 


1 


2% 




100% 


1-6* 


26 62% 


14 


33% 


2 


5% 




0% 




0% 




1 00% 


ii-r 


24 


57% 


14 


33% 


3 


7% 


1 


2% 




0% 




1 00% 


li-2 


22! 


52% 


1 1 


26% 


8 


19% 


1 


'2% 




0% 




100% 


11-3 


17p 


40% 


17 


40% 


■■ 5 


12% 


3 


7% 




I 0% 




100% 


iT-4* 


22^ 


52% 


■ {4 


33% 


6 


' 14% 




0% 


1 


0% 




1 00% 


11-5 


21 


50% 


16 


38% 


_ 3 


7% 


1 


2% 


1 


1 2% 




1 00% 


11-6 


39 


93% 


3 


7% 




0% 




0% 


I 


1 0% 




1 00% 


IIM 


2iT 


50% 


18 


43% 


3 


7% 




0% 




0% 




1 00% 


HI-2 


24; ' 


' 57% 


17 


”40% 


1 


2% 




0% 




0% 




100% 


111*3* 


3 ll_ 


_74% 


10 


__24% 




0% 


1 


2% 




0% 




100% 


MI-4 


29 


69% 


1 2 


29% 


" 1 


2% 




0% 




0% 




100% 


lli-5* 


27] 


”64% 


12 


’ '29% 


" 3 


7% 




0% 




0% 




100% 


lil-6 


19j 


45% 


18 


43% 


5 


12% 




0% 




0% 




I007p 


IV-1 


24 


57% 


14 


3_3% 


3i 


7% 


1 


2% 




0% 




100% 


iv’-2* 


"25^ 


60% 


16 


38% 


1 


'“2% 




0% 




0% 




1007o 


IV-3 


24, 


57% 


14 


33% 


3 


7% 


1 


2 % 




0% 




1 00% 


IV-4* _ 


24, 


57% 


15 


36% 


” 3’ 


7% 




0% 




_ 0% 




I007o 


IV-5~ 


2tl 


50% 


19 


_ 45% 


2 


5% 




6% 




0% 




1 00®/o 


IV-6 


2C 


50% 


17 


40% 


_ . ^ 


10% 




0% 




0% 




1007o 


Average 


23.2' 55.3% 


14.9 


35.5%|| 3.2 


7.5%1 


0.6 


1 .5% 


0.1 


0.2% 




lioo% 
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Appendix G 

Subject Judgement Consistency 



1 Subj. 


iO 


± 1 


± 2 


±3 


±4 


1 Total 


A1 


ii_L 


46% 


8 


33% 


3 


13% 


2 


8% 




0% 


; 1 00 % 


A2 


tAi#46% 


1 1 


46% 


2 


8% 




0% 




6% 


’ 1 00 % 






71% 


4 


17% 


3 


13% 




0% 




0% 


100 % 


10: 


42% 


1 0 


42% 


2 


8% 


2 


.8% 




0% 


|ioo% 


A5 


1_2^ 


50% 


9 


38% 


3 


i 3% 




' 0% 




0 % 


J00% 


A6 


. IO 4 . 


42% 


14 


58% 




0% 




0% 




1 ’ 0% 


’ 100% 


A7 


. 12^ 


50% 


5 


21% 


4 


' 17 % 


2 


8% 


1 


4% 


1 00% 


A8 


10^ 


42% 


12 


50% 


2 


8% 




0% 




0% 


100% 


A9 





58% 


6 


25% 


4 


' 17 % 




0% 




0% 


*100% 


A10 


1 0 


42% 


9 


38% 


5 


21% 




0% 




0% 


1 0.0% 


A11 


16; 


67% 


7 


29% 


1 


4% 




0% 




0% 


100% 


A12 


12 ' 


50% 


9 


38% 


2 


8% 


1 


4% 




0% 


■ 100% 


A13 


1 9: 


79% 


5 


21% 




6% 




0% 




0% 


100% 


B1 


15i 


63% 


9 


38% 




0% 




0% 




0% 


_ 1 00% 


B2 


. Jl. 


67% 


7 


29% 


1 


4% 




0% 




0% 


’ 100 % 


B3 


14.. 


58% 


8 


33% 


2 


8% 




0% 




0% 


100% 


B4 


13: 


54% 


8 


33% 


3 


13% 




6 % 




0% 


j’ 00 % 


B5 


10 


42% 


14 


58% 




0% 




0% 




0 % 


J00% 


B6 ' 


16^ 


"67% 


8 


33% 




0% 




0% 




0% 


. 1 00% 


B7 


18^ 


’ 75% 


' 6 


25% 




0% 




" 0 % 




0% 


■ 1 00% 


B8 


>7 


" 33% 


’ 1 2 


' 50 % 


4 


17% 




0% 




0% 


* 1 00% 


B9 


15 


63% 


5 


21 '% 


r 3 


1 3% 


1 


4% 




0% 


’ 1 00% 


B10 


Vi' 


46% 


■ "9 


' 38% 


'3 


13% 


1 


4% 




0% 


‘ 1 00% 


Bl’l 


' ' 1 sL 


"63% 


9 


”38% 




0% 




0% 




0% 


* 1 00% 


B12 


1 . 1 , 


46% 


12 


"" 50 "% 


1 


4% 




0% 




0% 


' 1 00% 


B13 


I’er 


67% 


.7 7 


29% 


1 


4% 




0% 




0% 


1 00% 


B14 


10^ 


42% 


1 10 


42% 


3 


7 13% 


1 


4% 




0% 


1 00% 


B15 


1 3' 


54% 


10 


" 42 % 


" 1 ’ 


4 % 




0% 




0% 


100% 


B16 


1 §! 


67% 


4 


77% 


4 


17% 




0% 




0% 


i 100% 


B17 


1 \ 


58% 


8 


33% 


' 2 


8% 




0% 




0% 


100% 


B18 


16: 


67% 


8 


" 33 % 




0% 




0%. 




0% 


’ 1 00% 


B19 


i'7i. 


71% 


“ -4 


17% 


2 


8% 


1 


4% 




0% 


' 1 00% 


B20 


1 4 ! 


'58% 


■ 9 


38% 


' 1 


4% 




0% 




0% 


‘ 1 00% 


B21 


7f 


’ 29% 


14 


58% 


~ 3 


13% 




0% 




0% 


’ 1 00% 


B22 


IV 


46% 


9 


38% 


3 


13% 




0% 


1 


4% 


’ 1 00% 


B23 


14; 


~58% 


? 


38% 


1 


4% 




0% 




0 % 


* 1 00% 


B24 ' 


13 ! 


' 54% 


" 8 


33% 


2 


8% 


1 


4% 




0% 


100% 


B25 


13’ 


54% 


10 


" 42 % 


1 


4% 




0% 




0% 


100% 


B26 


13^ 


' ' 54 % 


“9 


'38% 


1 


4% 


1 


4% 




0% 


_J00% 


B27 


14 ’ 


'58% 


7 


29% 


1 


4% 


2 


8% 




0% 


7~100% 


B28 




63% 


7 


29% 


2 


8% 




0% 




0% 


i 1 pp% 


B29 


15) 


63% 


9 


38% 




■■ 0% 




’' 0 % 




0% 


; 100% 


Average 


12.6' 


52.6% 


8.4 


34.9% 


2.4 


9.9% 


0.5 


2.2% 


0.1 


0.3% 


1 00% 
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Appendix H 
Order Effect 






^ '^liatements 


± 0 


± 1 


± 2 


± 3 


± 4 


Total of Differenceft 


1 -1 to 6 


40 


27 


7 


3 


1 


54 


ill i1Jo6 .. 


.42 


’30 


5 


1 




43 


11-1 lost 


■44 


'21 


1 1 


2 




49 


IV - 1 to 6t 


38 


31 


B 


1 




50 



t Orders of statemer}ts were randomized 
within Scenario II and iV in post-test 



tt Culculated by multiplying point of difference (l.e. 0, 1, 2, 3, or 4) 
by its number of occurence and then summing the total of point 
of difference, e.g., The total of difference for I - 1 to 6 is 
SUM(40*0+27*1+7*2+3*3+1 *4) 
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